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PREFACE 


This  document  contains  the  text  of  a  keynote  address  originally  given  by  Dr.  Barry  M.  Horowitz.  President 
and  Chief  Executive  Officer  of  The  MITRE  Corporation,  at  the  MITRE-Bedford  Software  Center  seminar 
Using  Off-the-Shelf  Software  in  O'  Systems. 
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INTRODUCTION 

The  use  of  commercial  off-the-shelf  (COTS) 
products  in  the  development  of  command,  control, 
communications,  and  intelligence  (Clt  systems  for 
the  Department  of  Defense  (DOD)  is  crucial.  Sig¬ 
nificant  reduction  in  the  DOD  budget  and  th„ 
expectation  of  further  reductions  will  force  the  de¬ 
fense  community  to  build  lower  com  sv  stems  that 
do  not  depend  heavily  on  customi/ed  hardware  and 
software.  The  concept  of  "open  system  architec¬ 
ture."  where  individual  subsystems  designed  and 
manufactured  by  different  industrial  organizations 
can  be  integrated  into  systems  of  high  performance 
(by  v  irtue  of  common  interface  standards),  is  mak¬ 
ing  it:,  appearance  in  the  commercial  marketplace. 
This  creates  the  possibility  that  large-scale  military 
systems  can  he  integrated  at  low  cost  from  commer¬ 
cial  products,  with  relatively  short  development 
schedules.  The  DOD  will  need  to  exploit  this 
possibility.  _ 

An  additional  impetus  for  the  use  of  commer¬ 
cial  products  is  the  public's  (and  military  com¬ 
manders")  growing  familiarity  with  products  readily 
available  on  the  market  for  home  and  business  use. 
As  part  of  their  personal  lives,  military  decision¬ 
makers  are  becoming  aware  of  the  powerful  com¬ 
puting  and  software  options  that  can  be  obtained 
off-the-shelf,  and  will  be  unwilling  to  believe  that 
custom  development  is  necessary  to  achieve  desired 
performance.  As  a  result,  it  is  realistic  to  predict 
that  commercial  systems  will  become  the  norm  for 
military  C  l  systems,  with  custom-designed  equip¬ 
ment  the  exception. 

Over  the  last  few  years,  substantial  effort  has 
been  spent  in  initial  attempts  to  integrate  commer¬ 
cially  based  products  into  military  Cl  sy  stems. 

F  rom  these  efforts,  the  defense  community  has  de¬ 
veloped  a  style  anil  approach  to  COTS  integration. 
Rapid  prototyping  is  being  employed  extensively  as 
part  ot  the  design  process.  Many  professionals  in 
the  prototype  arena  are  knowledgeable  about  open 
system  standards:  this  was  not  true  as  recently  as 
four  or  five  years  ago.  Much  more  attention  is  being 
given  to  commercial  standards,  because  they  are  be¬ 
coming  useful,  and  because  organizations  in  the 


commercial  world  have  been  formed  to  implement 
them. 

PARTICIPANTS  AND  THKIR  ROLFS 

There  are  three  major  participants  in  a  typical 
military  development:  ( I )  the  using  command,  or 
user,  who  sets  the  system  requirements  and  who  has 
final  responsibility  tor  operating  the  svstem:  (2)  the 
development  command,  or  developer,  who  is  given 
responsibility  for  developing  a  system  meeting  the 
user's  requirements:  and  t3)  industrv.  who  actually 
designs,  produces,  and  tests  both  the  development 
versions  and  the  final  production  versions  of  the 
desired  system.  However,  when  the  developments 
are  software-intensive  systems  with  an  emphasis  on 
the  use  of  COTS  products,  these  typical  roles  a.e 
not  always  followed.  Several  examples  are  illus¬ 
trated  in  the  following  list  of  projects,  for  which  the 
electronic  Systems  Division  (HSD)  of  the  Air  Force 
•v  as  the  development  command,  supported  by  The 
MITRF  Corporation.  The  development  approach, 
and  the  roles  apportioned  to  the  three  participants, 
were  somewhat  different  in  each  case  (see  table  1 ). 

•  Sentinel  By  te's  mission  is  to  make  intelligence 
data  originating  from  higher  lev  els  of  command 
av  ailable  to  the  pilot  doing  mission  planning. 

•  Granite  Sentry  is  a  command  post,  located  in  the 
Cheyenne  Mountain  Complex  in  Colorado,  for 
assessing  ballistic  missile  warnings  from  our 
strategic  radar  and  infrared  sensors. 

•  NORTIC  is  a  command  post  designed  to  help 
NORAD  track  drug  smugglers. 

•  The  Message  Handling  System  receives  mes¬ 
sages  on  a  network  and  disseminates  them  on  the 
basis  of  full  profiling. 

•  The  Airborne  Command  and  Control  Center 

( ABCCCt  is  an  airborne  platform  that  carries 
operators  and  a  large  varietv  of  radios  for  com¬ 
municating  during  a  crisis. 

•  The  Mission  Planning  System  is  an  automated 
aid  to  assist  in  the  planning  of  air  missions. 


I 


Table  I.  Kxamples  of  KSD  as  Development  Command  on  Software-Intensive  Systems 


System 

l  ser 

User's  Role 

Developer’s  Role 

Industry’s  Role 

Sentinel  Byte 

Li  SAFE. 
PACAF. 
TAC. 

MAC. 

SAC 

Gene  rated  rec|  u  i  remen t  s 

Built  two  COTS 
prototy  pes  installed  at 

AF  sites.  Success  led  to 
decision  to  buy  15  more 

Replicated  dev  eloper's 
prototypes 

Granite  Sentry 

NORAD 

Hired  and  managed 
support  contractors  to 
develop  system 

Generated  and  analy  /ed 
system  architecture, 
maximizing  COTS  use 

Provided  software 
support  to  user 

NORTIC 

NORAD 

Generated  requirements 

Developed  system 
architecture  and  system 
design.  Selected  a 
contractor  to  buy  COTS 
components.  Selected 
another  support  contrac¬ 
tor  to  integrate,  test,  and 
maintain  sy  stem 

Bought  COTS  compo¬ 
nents  to  implement 
de  v  e  1  ope r "  s  a rc  h  i t ec t  u re . 
Integrated,  tested,  and 
maintained  system 

Message 

Handling 

System 

DIA 

Generated  requirements 

Generated  specification. 
Hired  contractor  to 
demonstrate  initial 
prototype 

Designed  and  built 
prototype,  maximizing 
COTS  use 

ABCCC 

TAC 

Generated  requirements 

Generated  product 
specification.  Selected 
two  contractors  to  build 
prototypes 

Designed  and  built 
prototypes  and 
production  version 

Mission 

Planning 

Sy  stem 

TAC 

Built  3/4  of  system, 
asked  developer  to 
complete 

Transitioned  sy  stem  to 
more  open  architecture, 
so  that  capability  could 
grow  by  addition  of 
COTS  products 

Designed  and  built 

sy  stem,  using 

tie \  e  1  ope r ' s  arc h i tectu re 

JUDGMENT 


The  problem  with  using  commercial  products 
is  relating  the  commercially  available  equipment 
and  software  to  the  mission  needs.  This  problem 
involves  the  different  skills  and  judgments  listed  in 
table  2. 


Table  2.  Issues  and  Judgments  Regarding 
Commercially  Available  Equipment 
and  Software 


Issue 

Kind  of 
Judgment 

W'hat  products  are  available 
on  the  market? 

Oversight  of 
entire  commercial 
market 

Can  commercial  products  be 
integrated,  even  if  indiv  idual 
products  have  adequate 
capability? 

Technical 

judgment 

Will  the  completed  COTS 
system  he  operationally 
useful.’ 

User  judgment 

How  much  w  ill  it  cost  to  add 
capability  later  ’ 

Technical 

judgment 

How  difficult  w  ill  the  COTS 
system  be  to  modify? 

Technical 

judgment 

To  what  extent  is  the  user 
w  illing  to  compromise  on 
the  system  requirements, 
balancing  w  hat  is  desired 
versus  what  is  readily 
attainable? 

Mission  judgment 

IMPORT  ANT  PHILOSOPHIC  AL  ISSUES 

In  each  of  the  projects  described,  different 
judgments  were  made  about  the  mixture  of  roles 
which,  when  coupled  with  business  decisions,  led  to 
different  acquisition  styles. 

While  there  is  no  best  acquisition  style,  the  dif¬ 
ferent  styles  used  in  the  projects  described  above- 
arc  not  a  sign  of  healthy  variety.  They  did  not  result 


from  a  set  of  logical  decisions,  but  instead  were 
driven  by  the  vary  ing  political  and  economic  envi¬ 
ronments  surrounding  each  development.  No  clear 
method  or  policy  is  available  to  aid  in  the  selection 
of  acquisition  approaches.  The  defense  community 
has  not  yet  dealt  adequately  with  some  important 
philosophical  issues  regarding  developments  incor¬ 
porating  commercial  products.  Two  of  these  issues 
relate  to  performance  specifications  and  sy  stem 
architecture. 

Performance  Specifications 

The  problem  of  performance  specifications  in 
cases  where  COTS  use  is  desired  has  not  yet  been 
satisfactorily  resolved.  Air  Force  system  develop¬ 
ments  are  traditionally  begun  by  writing  perfor¬ 
mance  and  functional  specifications.  These 
specifications  explain  vvliat  the  government  wants  a 
contractor  to  do.  thereby  prov  iding  the  contractor 
with  a  firm  basis  on  which  to  accept  the  risk  of 
building  the  hardware  and  software  for  the  system 
at  an  estimated  cost. 

However,  when  a  system  is  to  be  developed 
using  commercial  products,  the  design  latitude 
av  ailable  when  compared  to  building  a  sy  stem  of 
custom  components  is  severely  constrained.  Design 
freedom  is  implicit  in  the  idea  of  a  specification. 

For  programs  driven  by  the  desire  to  integrate  un¬ 
modified  commercial  hardware  and  software  prod¬ 
ucts.  freedom  of  design  is  a  false  assumption.  The 
concept  of  writing  a  specification  does  not  fit  with 
the  idea  of  building  sy  stems  from  commercial  parts. 

In  principle,  a  conflict  between  specifications 
and  the  use  of  COTS  products  is  not  necessary. 
Before  making  a  bid.  industry  could  perform  the 
trade-off  studies  needed  to  identify  applicable 
COTS  products,  select  those  that  represent  the  best 
match  to  the  specification,  and  verify  that  the  se¬ 
lected  products  could  actually  be  integrated  within 
the  proposed  schedule  and  cost.  However,  projects 
based  on  commercial  products  virtually  always 
have  small  budgets,  and  it  is  unreasonable  to  expect 
a  company  to  invest  in  extensive  trade-off  studies 
before  it  bids  on  a  project  that  otters  only  a  modest 
return. 
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In  practice,  industry  must  use  its  knowledge 
about  commercial  components  (which  currently  is 
limited)  to  make  a  calculated  guess  about  whether 
those  products  can  be  integrated  to  meet  the  specifi¬ 
cations.  A  wrong  guess  results  in  no  options  for  re¬ 
covery.  because  typically  no  money  has  been 
budgeted  by  the  government  for  any  custom  design, 
for  example,  some  development  projects  at  KSD 
have  been  cancelled  because  the  efforts  were  started 
w  ith  a  specification  anti  were  later  discovered  to  be 
infeasible  using  the  selected  COTS  products:  both 
industry  and  the  government  took  losses.  A  way 
must  be  found  to  remove  specifications  from  the 
process  of  buy  ing  COTS-based  systems. 

System  Architecture 

A  second  problem  not  adequately  resolved  is 
system  architecture,  which  affects  life-cycle  costs  in 
software-intensive  systems.  A  large  percentage  of  a 
software  system  is  changed  over  its  life  cycle,  and  a 
large  fraction  of  the  cost  of  owning  a  softw  are  sy  s¬ 
tem  occurs  after  development.  Various  estimates 
indicate  about  60  percent  of  the  cost  of  software  is 
incurred  after  it  is  shipped  to  the  user.  Of  that  trac¬ 
tion.  about  70  percent  is  spent  adding  new  features, 
either  because  the  original  capability  was  unsatis¬ 
factory  or  the  mission  changed.  Similarly  ,  at  KSD. 
about  half  the  software  effort  is  spent  reworking 
what  was  initially  developed,  even  during  develop¬ 
ment.  In  total,  about  SO  percent  of  C'l  software 
development  efforts  are  spent  responding  to 
changes  in  the  original  specifications.  The  nature  of 
('M  dev  elopment  precludes  holding  firm  to  specifi¬ 
cations  over  a  long  lime,  due  to  the  rapid  tempo  of 
new  commercial  developments. 

!  o  accommodate  such  variability  in  require¬ 
ments.  the  system  architecture  must  be  designed  to 
make  applications  readily  changeable.  There  is  no 
point  m  specifying  every  system  lunction  in  exact 
detail,  because  most  functions  will  undoubtedly 
change  somewhat  during  the  development.  Without 
an  architecture  that  supports  changes,  substantial 
development  fund'  will  be  spent  changing  functions 
that  have  been  optimized  to  the  wrong  goal. 


This  situation  arises  because  the  using  com¬ 
mands  believe  they  know  exactly  what  is  needed  at 
the  beginning  of  the  development.  The  development 
commands  believ  e  their  job  is  to  sell  the  user's  de¬ 
fined  requirements.  Thus,  the  development  process 
does  not  promulgate  any  scheme  for  thinking  aboui 
adjustments.  In  fact,  the  current  process  discourages 
thinking  about  adjustments  because  success  is  tie- 
lined  as  meeting  the  specification. 

rkdtcim;  dependence  on 

SPECIFICATIONS 

The  problem  of  changing  specifications  can  be 
addressed  by  creating  a  new  process,  beginning 
with  a  team  whose  only  job  is  to  devise  an  architec¬ 
ture  that  will  support  making  the  system  change¬ 
able.  With  a  sound  architectural  basis,  even  if  the 
team  makes  many  errors  building  the  functions,  it 
will  have  created  a  scheme  that  makes  it  inexpen¬ 
sive  to  nake  changes.  Today,  most  of  the  effort 
goes  into  looking  at  the  user  display  s  and  the  func¬ 
tions  to  be  sure  they  are  correct,  rather  than  looking 
at  whether  or  not  the  architecture  is  adjustable. 

So  far  as  I  am  aware,  the  government  has  never 
initiated  a  C  l  development  effort  by  asking  for  a 
system  architecture  explicitly  designed  for  change¬ 
ability.  and  it  has  never  asked  that  a  given  architec¬ 
ture  be  evaluated  for  its  ability  to  support  change. 
No  method  exists  lor  generating  such  a  specifica¬ 
tion.  and  system  engineers  have  no  satisfactory  way 
to  evaluate  architectures.  Our  community  must  in¬ 
vent  new  techniques  in  this  area.  We  should  first 
recognize  that  the  development  of  a  C  I  system  is 
aimed  at  a  changeable  architecture  and  a  first 
implementation  of  function  -  and  then  invest  vig¬ 
orous  effort  in  the  design  of  the  architei  :ure  and 
perhaps  equal  effort  in  the  subsequent  functional 
design  (but  not  much  more  effort  in  (lie  lunctional 
design,  as  is  done  todav  ). 

A  parallel  may  be  drawn  with  the  housing  in¬ 
dustry.  Building  a  house  involves  a  very  mature 
technology,  with  two  major  aspects:  (  I  i  the  finish 
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ing  of  the  house  interior,  concerning  details  such  as 
the  number  of  baths  and  the  dimensions  of  the 
kitchen,  and  (2)  the  basic  architectural  design  of  the 
house. 

These  two  aspects  of  house  building,  finishing 
and  architecture,  are  handled  by  two  different  sets 
of  professionals.  Inspectors  are  employed  to  make 
certain  that  state  and  local  building  codes  are  satis¬ 
fied.  Even  though  the  consumer  may  not  have  tech¬ 
nical  knowledge  about  the  spacing  of  studs  or 
thickness  of  concrete,  he  or  she  knows  it  is  prudent 
to  have  an  expert  thoroughly  check  these  and  other 
building  parameters.  Few  home-buyers  would  pur¬ 
chase  a  house  that  did  not  have  a  careful  review  by 
a  building  codes  expert. 

With  confidence  that  the  house's  basic  archi¬ 
tecture  is  sound  and  therefore  extendable  and 
changeable,  the  consumer  can  specify  the  details  of 
internal  functions  with  the  knowledge  that  those 
functions  are  likely  to  change  as  economic  circum¬ 
stances  change:  for  example,  the  consumer  may 
someday  w  ant  bigger  bathrooms  or  a  larger  kitchen. 

In  software  systems,  by  analogy,  the  detailed 
functions  inside  the  house  are  emphasized  while  the 
foundation  of  the  house  is  virtually  ignored.  Al¬ 
though  the  users  are  fully  knowledgeable  about  the 
desired  functions  of  the  software  system,  they  have 
little  skill  in  the  "building  codes"  for  software.  Un¬ 
fortunately,  unlike  the  housing  consumer,  the  soft¬ 
ware  user  as  yet  has  no  orientation  toward  bringing 
in  the  equivalent  of  the  building  codes  inspector 
who  can  insist  that  the  basic  software  architecture 
be  sound  and  readily  accommodate  change  in  the 
detailed  functions. 

Performance  specifications  that  focus  on  com¬ 
pleting  an  architecture  create  a  negative  environ¬ 
ment  for  building  systems  from  commercial 
products,  and  a  negative  environment  for  looking 
at  the  architecture  from  the  viewpoint  of  change¬ 
ability.  How  might  complex  CM  software  systems 
be  developed  from  commercial  products,  using  a 
process  that  puts  more  emphasis  on  software  archi¬ 
tecture  and  is  not  based  on  performance  specifica¬ 
tions?  One  approach,  described  below,  is  to  form  a 


team  consisting  of  the  users,  developers,  and  indus¬ 
try  in  various  combinations,  depending  .  n  the  de¬ 
velopment  phase. 

Understand  the  User's  Needs 

The  team,  which  could  consist  of  a  mixture  of 
users  and  developers,  must  understand  the  user's 
desires  in  detail. 

Understand  COTS  Products 

The  team  must  have  detailed  knowledge  of  as 
many  available  commercial  products  as  possible. 
There  are  countless  commercial  products  on  the 
market,  and  new  ones  are  appearing  at  a  tremen¬ 
dous  rate.  A  substantial  effort  is  needed  to  under¬ 
stand  available  products. 

It  is  not  practical  to  set  up  a  special  team  that 
would  become  expert  in  all  the  commercial  prod¬ 
ucts  because  the  team  would  be  overwhelmed  by 
the  volume  of  products.  The  only  feasible  method  is 
to  form  one  or  more  teams  that  work  on  and  over¬ 
see  many  different  implementations  of  CM  systems. 
In  that  way.  knowledge  would  be  gained  about  not 
only  what  products  are  available,  but  also  how  well 
they  integrate  and  what  problems  occur  with  par¬ 
ticular  integrations.  This  knowledge  would  not  be 
comprehensive,  but  over  time  would  span  a  large 
fraction  of  the  necessary  ingredients  in  CM  design. 

Learn  How  to  Evaluate  System  Architecture 

Satisfactory  methods  for  evaluating  architec¬ 
tures  must  be  developed.  Two  relatively  crude  ap¬ 
proaches  are  occasionally  employed  now.  The  first 
involves  building  prototypes.  According  to  my  ex¬ 
perience.  the  general  interest  in  prototypes  tends  to 
be  oriented  to  display,  analogous  to  selecting  the 
kind  of  kitchen  the  consumer  wants  in  his  or  her 
house.  The  second  method  involves  evaluating  pro¬ 
totypes  with  respect  to  their  architectures.  The  sec¬ 
ond  method  is  as  crucial  to  the  user,  but  is  not  as 
apparent.  As  a  result,  there  are  few  funded  efforts 
devoted  to  evaluating  the  architectures  of  proto¬ 
types.  Unfortunately,  funding  that  could  be  used  for 
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this  purpose  is  often  expended  on  fixing  systems 
that  have  not  been  properly  developed. 

An  important  aspect  of  architecture  evaluation 
is  a  method  for  rejecting  products  that  seem  attrac¬ 
tive  but  cannot  be  extended  or  expanded  within  the 
chosen  architecture.  There  is  a  strong  tendency  to 
incorporate  a  product  if  it  appears  to  offer  some  at¬ 
tractive  new  feature,  w  ithout  careful  examination  to 
see  whether  the  product  will  allow  functional 
changes  at  a  later  date,  and  without  an  estimate  of 
the  cost  to  add  on  the  next  new  commercial  capabil¬ 
ity.  Although  prototypes  can  help  in  estimating  the 
cost  of  future  changes,  they  are  seldom  used  in  this 
fashion,  even  though  the  cost  issue  is  crucial. 

COTS-based  C'l  developments  tend  to  be  low- 
cost  projects,  driven  by  time  and  budget.  If  the  de¬ 
velopment  cost  and  the  user  s  budget  were  know  n 
at  the  outset,  an  informed  judgment  could  be  made 
about  what  functional  capabilities  are  affordable 
(whereas  a  specification  must  be  generated  at  a  time 
when  the  feasibility  of  using  COTS  products  is  in 
doubt,  as  well  as  the  practicality  of  future  addi¬ 
tions).  A  better  process  would  be  to  build  a  proto¬ 
type  capability,  leam  about  cost  and  architecture 
from  the  prototyping  exercise,  discuss  functional 
performance  options  and  their  associated  cost  with 
the  users,  and  then,  given  a  budget,  decide  what  is 
possible. 

Bring  in  Industry  Differently 

When  the  time  arrives  to  bring  industry  into  the 
team,  a  new  method  must  be  found  to  make  the  se¬ 
lection  from  among  the  bidders  for  a  COTS  C’l  de¬ 
velopment.  Often,  the  company  selected  by  the 
government  is  the  least  expensive  and  the  most 
readily  available,  because  both  funds  and  schedule 
are  severely  constrained.  With  reference  to  the 
house-building  analogy,  it  is  clear  that  we  would 
not  w  ant  to  choose  the  contractor  for  our  home 
solely  on  the  basis  of  cost  and  availability. 

Having  chosen  the  basic  COTS  elements  of  the 
architecture,  a  rational  approach  is  to  favor  compa¬ 
nies  that  know  the  most  about  that  class  of  COTS 
products,  because  it  is  reasonable  to  expect  those 


companies  to  be  more  successful  at  integrating  the 
products  at  the  lowest  cost.  In  a  given  development, 
if  one  company  were  found  to  be  most  knowledge¬ 
able  about  all  of  the  applicable  COTS  products, 
then  it  would  seem  appropriate  to  select  that  com¬ 
pany.  On  the  other  hand,  following  this  reasoning, 
if  the  selected  architecture  called  for  the  integration 
of  COTS  elements  A.  B.  C.  and  D.  and  if  company 
I  was  found  to  be  expert  in  integrating  items  A  and 
B  while  company  2  was  expert  in  integrating  items 
C  and  D.  the  government  might  choose  hath  con¬ 
tractors.  Unfortunately,  the  government  would  be 
afraid  to  take  this  latter  course,  because  in  effect  the 
government  would  be  acting  as  the  prime  contractor 
responsible  for  integrating  the  system  and  would 
therefore  be  accountable  for  successful  completion 
of  the  overall  development. 

However,  the  problem  of  government  account¬ 
ability  may  not  be  severe.  The  government's  atti¬ 
tude  toward  accountability  stems  from  custom- 
development  projects  that  typically  have  very  high 
cost  (tens  to  hundreds  of  millions  of  dollars);  the 
fear  of  becoming  entangled  in  expensive  litigation 
is  understandable  in  such  cases.  By  contrast,  the 
scale  of  COTS  Cl  developments  is  typically  in  the 
range  of  four  or  five  million  dollars.  When  the  gov  ¬ 
ernment  team  has  skills  and  knowledge  to  do  the 
integration  job  more  professionally  than  an  indus¬ 
trial  team  (who  may  not  know  the  user  and  product 
selection  as  well),  the  government  should  strongly 
consider  taking  on  the  integrator  role. 

RELATION  OF  PRESENT  METHODS 
TO  INDUSTRY 

While  industry  might  not  agree.  I  believe  our 
current  practices  in  the  development  of  C’l  COTS- 
based  systems  are  harmful  to  industry.  A  typical 
scenario  illustrates  the  point  when  the  development 
command  requests  proposals  for  a  C‘l  system  built 
primarily  from  COTS  products.  This  request  is 
made  even  though  there  is  no  proof  that  the  specifi¬ 
cation  can  be  met  with  COTS  products.  (Occasion¬ 
ally.  feasibility  is  asserted  by  the  developer  because 
one  COTS  version  of  the  desired  system  has  been 
prototyped.  However,  the  selected  contractor  is  free 


to  choose  any  set  of  COTS  products.  The  developer 
does  not  know  whether  the  conmic tor's  selection  ot' 
products  can  do  the  job.  and  neither  does  the  con¬ 
tractor.  for  the  reason  stated  earlier.  No  contractor 
can  afford  to  invest  in  proving  feasibility  before  the 
bid  is  made.)  After  the  selection,  the  contractor  be¬ 
gins  the  development,  w  ith  the  understanding  that 
the  specification  must  be  met.  When  the  contractor 
is  unable  to  meet  the  specification  within  schedule, 
the  contractor  begins  to  lose  money.  Because  these 
jobs  are  tightly  budgeted,  it  takes  only  a  few 
months  of  extra  time  before  the  contractor's  profits 
vanish.  The  results  of  extended  court  contests  are 
long  delay  s,  increased  costs,  and  undesirable  prod¬ 
ucts.  Industry  bears  the  risk  of  meeting  the  user's 
requirements  w  ith  its  own  selection  of  off-the-shelf 
products. 

HIGH-RISK  AND  LOW-RISK  PHASES  OF 
DEVELOPMENT 

Once  the  trade-offs  of  meeting  user  desires  ver¬ 
sus  the  performance  of  COTS  products  have  been 
made  and  are  thoroughly  understood,  the  develop¬ 
ment  of  COTS-based  C  l  systems  becomes  low 
risk.  The  trade-offs  allow  accurate  estimation  of 
integration  costs  and  of  the  cost  to  incrementally 
add  capability  to  the  system.  When  the  user's  real 
budget  is  known,  and  when  the  user  is  willing  to 
temper  his  or  her  requirements  to  stay  w  ithin  that 
budget,  the  risk  becomes  small.  Generally .  very  ma¬ 
ture  C’OTS  software  is  used  so  that  the  risk  of  en¬ 
countering  severe  software  bugs  is  relatively  low. 

However,  before  the  trade-offs  have  been  com¬ 
pleted,  the  development  risk  is  very  high  because 
the  contractor  is  expected  to  meet  a  rigid  set  of  re¬ 
quirements  but  is  not  allowed  (oi  cannot  afford)  to 
do  custom  design.  If  the  government  were  to  select 
the  products  and  take  the  risk  that  the  correct  match 
to  the  user's  needs  has  been  made,  then  industry 
could  implement  the  design  without  being  account¬ 
able  for  the  match  to  user  requirements,  and  the  risk 
to  industry  would  be  considerably  less.  Industry 
should  take  responsibility  for  integrating  the  COTS 
products  well,  and  should  not  be  concerned  about 


whether  the  product  selection  is  optimum  for  the 
user.  The  government  should  be  responsible  for  en¬ 
suring  the  product  selection  matches  user  needs. 

A  NEW  DEVELOPMENT  APPROAC  H  FOR 
COTS-BASED  C'l  SYSTEMS 

I'he  preceding  discussion  results  in  a  different 
approach  to  the  development  of  COTS-based  C  l 
systems: 

•  The  development  command  takes  responsibility 
for  ensuring  that  three  critical  aspects  of  the  de  ¬ 
velopment  are  properly  traded  off  with  respect  to 
cost  and  performance: 

-  Balance  of  the  cost  to  build  and  modify 

-  Availability  of  products 

-  Desires  of  the  user 

•  The  development  command  takes  responsibility 
for  deciding  w  hether,  in  a  given  case,  it  is  better 
to  integrate  a  system  with  a  single  contractor  or 
multiple  contractors. 

•  Even  if  a  single  contractor  is  selected,  the  user 
and  development  commands  take  : esponsibility 
for  correct  product  selection.  Contractor  selec¬ 
tion  is  not  based  on  specifications,  but  on  the 
contractor's  ability  to  integrate.  Under  this  new 
procedure,  if  the  performance  of  the  integrated 
system  is  poor,  then  the  assessment  of  blame 
depends  on  the  cause.  If  both  the  contractor  and 
the  integration  plan  prove  to  be  sound,  then  the 
development  command  is  at  fault.  If.  however, 
the  integration  was  not  performed  well,  then  the 
contractor  is  at  fault  and  should  be  replaced. 

•  The  development  command  must  have  a  reason¬ 
ably  comprehensive  knowledge  of  the  COTS 
marketplace  to  ensure  that  the  final  selection  of 
products  is  chosen  from  the  very  large  number 
of  commercial  products  available.  The  process 
requires  confidence  that  a  good  source  has  not 
been  overlooked,  or  that  some  different  COTS 
vendor  w  ill  not  later  confront  the  developer  with 
a  superior  product. 
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•  The  development  command  must  be  prepared  to 
evaluate  an  architecture  for  basic  soundness  and 
the  ability  for  change  at  reasonable  cost.  The 
user  must  be  convinced  that  the  selected  archi¬ 
tecture  is  resilient  and  adaptable.  The  govern¬ 
ment's  job  is  to  select  the  system  architecture 
that  satisfies  the  user's  requirements  at  a  time 
when  the  final  system  requirements  are  not  yet 
fully  know  n. 

•  Finally,  the  development  command  must  find  a 
way  to  judge  the  integration  activities  performed 
by  the  contractor.  Today,  this  issue  is  confused 
by  the  need  to  meet  specifications.  When  the 
contractor  has  conducted  a  high  quality  integra¬ 
tion  effort,  it  is  not  appropriate  to  hold  the  con¬ 
tractor  responsible  for  the  performance  of 
commercial  equipment  and  software. 

This  summarizes  a  new  model  of  how  COTS- 
based  developments  should  be  done.  It  is  important 
that  an  effective  approach  for  this  class  of  develop¬ 
ment  be  put  in  place,  because  the  economics  of  sys¬ 
tem  development  will  greatly  emphasize  the 
integration  of  COTS  products. 

LIFE-CYCLE  ISSUES 

We  are  entering  a  period  in  which  the  DOD 
will  not  have  enough  money  to  replace  C'l  systems: 
instead,  the  systems  will  have  to  be  maintained  and 
upgraded  over  a  period  of  many  years.  It  is  not 
clear  who  will  be  given  this  maintenance  role  — 
industry,  the  user,  the  developer,  or  the  logistics 
organizations.  The  issue  of  organizational  role  is 
important  and  should  not  be  dismissed  on  the  basis 
of  a  given  command's  mission  or  the  assertion  that 
the  user  must  retain  control  of  the  process. 


If  a  system  composed  of  COTS  products  is 
modified  or  evolutionary  improvements  are  made, 
the  integrated  off-the-shelf  products  of  the  sy  stem 
should  not  be  modified  because  the  cost  w  ill  be 
high.  Any  additions  should  be  more  off-the-shelf 
products,  and  perhaps  a  small  number  of 
custom-designed  modules. 

The  most  important  factor  in  upgrading  a  sys¬ 
tem  should  be  knowledge  of  the  system's  architec¬ 
ture.  If  a  system's  architecture  has  been  properly 
designed  to  accommodate  change,  the  people  who 
perform  the  life-cycle  maintenance  and  upgrade 
role  should  be  experts  in  that  architecture  so  they 
can  properly  exploit  the  opportunities  for  change  in 
the  system.  At  present,  this  assignment  of  experts 
does  not  take  place,  and  no  effort  is  expended  to 
provide  the  necessary  architectural  information  to 
the  actual  maintainers  and  upgraders.  When  indus¬ 
try  is  given  the  task,  the  people  who  do  the  mainte¬ 
nance  are  not  the  designers  of  the  architecture,  but 
rather  those  who  wrote  the  final  application 
software. 

The  life  cycle  is  something  that  we  are  not 
thinking  about  properly,  and  the  issues  of  transfers 
of  command,  transfers  of  responsibilities,  and  trans¬ 
fers  of  money  are  quite  complex.  However,  this  is 
an  area  where  we  must  create  a  more  efficient  way 
of  doing  business.  Ideas  for  handling  the  full  life- 
cycle  of  COTS-based  C’l  systems  need  develop¬ 
ment  to  exploit  the  opportunities  to  lower  risk  and 
save  money. 
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